Optimize managed postprocessing and allocations - #5
Closed
ericstj wants to merge 1 commit into
Closed
Conversation
Decode raw model logits, use sparse Viterbi transitions, and pool temporary inference and backpointer buffers. Add parity coverage and before/after benchmarks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a9621082-f9c8-4ec8-ac3c-faaf34eb7d0f
Owner
Author
|
Superseded by the same commit on the dedicated optimization branch. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR optimizes the managed post-processing path of PrivacyFilterNet inference by eliminating log-softmax from production decoding, introducing a sparse Viterbi predecessor representation with pooled backpointers, and pooling/copy-avoiding inference buffers to reduce allocations while preserving decode outputs.
Changes:
- Decode using raw logits (no production log-softmax) and add parity tests to ensure ArgMax/Viterbi invariance.
- Replace dense Viterbi transition matrix + int backpointers with sparse predecessor tables and pooled compact (byte) backpointers.
- Pool inference input/score buffers and avoid copying dense ONNX outputs when possible; expand benchmark coverage and document results.
Show a summary per file
| File | Description |
|---|---|
| tests/PrivacyFilter.Net.Tests/DecoderTests.cs | Adds randomized parity tests (logits vs log-softmax) and a dense reference decoder to validate sparse Viterbi correctness. |
| src/PrivacyFilter.Net/ViterbiDecoder.cs | Implements sparse predecessor tables and pooled compact backpointers; updates decoder API to use ReadOnlySpan<float>. |
| src/PrivacyFilter.Net/PrivacyFilter.Net.csproj | Exposes internals to the benchmarks project to enable benchmark/test access to internal helpers. |
| src/PrivacyFilter.Net/PrivacyFilter.cs | Pools inference buffers, avoids copying dense logits outputs, and decodes directly from logits using span-based APIs. |
| bench/results.md | Documents benchmark results for the managed postprocessing optimizations. |
| bench/PrivacyFilter.Net.Benchmarks/Program.cs | Adds managed postprocessing microbenchmarks and switches to BenchmarkSwitcher for multiple benchmark classes. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comments suppressed due to low confidence (1)
src/PrivacyFilter.Net/PrivacyFilter.cs:208
- Returning pooled
ids/maskarrays withclearArray: trueclears the entire rented buffers (often larger thanwindowLength), which can add avoidable overhead per inference window. Consider clearing only the used prefix and returning withclearArray: false(or add a comment if full-buffer clearing is a deliberate security choice).
finally
{
ArrayPool<long>.Shared.Return(ids, clearArray: true);
ArrayPool<long>.Shared.Return(mask, clearArray: true);
}
- Files reviewed: 6/6 changed files
- Comments generated: 2
- Review effort level: Low
Comment on lines
+117
to
+120
| finally | ||
| { | ||
| ArrayPool<float>.Shared.Return(scoreBuffer, clearArray: true); | ||
| } |
Comment on lines
+170
to
176
| finally | ||
| { | ||
| lastLabel = backpointers[(token * classCount) + lastLabel]; | ||
| path[token] = lastLabel; | ||
| if (backpointerBuffer is not null) | ||
| { | ||
| ArrayPool<byte>.Shared.Return(backpointerBuffer, clearArray: true); | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Results
Testing
dotnet test PrivacyFilter.Net.sln -c Release --nologo